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AUDITOR  GENERAL,  DEPARTMENT  OF  THE  ARMY 

SUBJECT:  Audit  Report  on  the  Use  of  Exit  Criteria  for  Major  Defense  Systems 
(Report  No.  D-2001-032) 


We  are  providing  this  report  for  review  and  comment.  The  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  did  not  respond  to  the  draft 
report;  however,  we  considered  comments  received  from  the  Deputy  for  Systems 
Acquisition,  U.S.  Army  Tank-automotive  and  Armaments  Command,  and  the 
Commander,  Space  and  Naval  Warfare  Systems  Command,  when  preparing  the  final 
report. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  resolved  promptly. 
We  request  that  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  respond  to  Recommendations  A.l.  and  B.  by  February  12,  2001. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 
should  be  directed  to  Mr.  John  E.  Meling  at  (703)  604-9091  (DSN  664-9091) 
(jmeling@dodig.osd.mil)  or  Mr.  Douglas  P.  Neville  at  (703)  604-9076 
(DSN  664-9076)  (dpneville@dodig.osd.mil).  See  Appendix  E  for  the  report 
^stribution.  Audit  team  members  are  listed  inside  the  back  cover. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 
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Use  of  Exit  Criteria  for  Major  Defense  Systems 
Executive  Summary 


Introduction.  Milestone  decision  authorities  use  exit  criteria  to  establish  goals  for 
major  Defense  programs  during  an  acquisition  phase.  Exit  criteria  are  program-specific 
accomplishments  that  program  managers  must  satisfactorily  demonstrate  before  a 
program  can  progress  further  in  the  current  acquisition  phase  or  transition  to  the  next 
acquisition  phase.  By  satisfying  the  exit  criteria,  the  program  manager  denionstrates  to 
the  milestone  decision  authority  that  a  program  is  on  schedule  to  achieving  its  final 
program  goals.  To  perform  the  audit,  we  selected  nine  major  Defense  programs  for 
review  that  the  program  offices  estimated  to  total  $78  billion  in  development  and 
procurement  costs.  The  milestone  decision  authority  for  three  of  the  major  Defense 
acquisition  programs  included  in  our  sample  was  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  while  the  milestone  decision  authorities  for  the 
remaining  six  programs  (three  Army,  one  Navy,  and  two  Air  Force)  were  the 
Component  Acquisition  Executives  or  their  designees. 

Objectives.  The  audit  objective  was  to  evaluate  milestone  decision  authorities’  use  of 
exit  criteria  to  track  program  progress  in  important  technical,  schedule,  and 
management  risk  areas.  Specificdly,  we  determined  whether  milestone  decision 
authorities  use  exit  criteria  to  track  acquisition  program  progress  in  meeting  program 
goals  and  to  aid  in  deciding  whether  programs  should  continue  within  an  acquisition 
phase  or  progress  to  the  next  acquisition  phase.  In  addition,  we  evaluated  the 
management  control  program  as  it  relates  to  the  audit  objective. 

Results.  Our  review  of  the  nine  major  Defense  programs  (three  Army,  three  Navy,  and 
three  Air  Force)  showed  that  improvements  were  needed  in  the  establishment  of  exit 
criteria  at  milestone  decision  points  and  in  reporting  the  status  toward  attaining  exit 
criteria  requirements  to  milestone  decision  authorities. 

•  For  seven  of  the  nine  programs  reviewed,  milestone  decision  authorities  did 
not  ensure  that  program  managers  proposed  program-specific  exit  criteria  for 
use  at  the  future  milestone  decision  point(s).  As  a  result,  the  milestone 
decision  authorities  were  limited  in  their  ability  to  use  exit  criteria  as  a 
management  tool  to  determine  whether  programs  under  their  review  and 
oversight  should  progress  within  an  acquisition  phase  or  continue  into  the  next 
acquisition  phase  at  milestone  decision  points  (finding  A). 
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•  Program  managers  for  three  of  the  five  major  Defense  acquisition  programs 
reviewed  did  not  report  their  status  toward  attaining  exit  criteria  requirements 
in  the  quarterly  Defense  Acquisition  Executive  Summary.  As  a  result, 
milestone  decision  authorities  and  Office  of  the  Secretary  of.  Defense  action 
officers  did  not  have  information,  through  the  Defense  Acquisition  Executive 
Summary,  to  use  as  a  management  tool  for  assessing  each  program’s  progress 
toward  satisfying  exit  criteria  requirements  and  for  providing  direction,  when 
needed,  between  milestone  decision  points  (finding  B). 

Recommendations  in  this  report,  if  implemented,  will  ensure  that  exit  criteria  are 
established,  that  the  status  of  approved  exit  criteria  is  routinely  reported  in  the  Defense 
Acquisition  Executive  Summary,  and  will  correct  the  material  management  control 
weaknesses  identified  in  Appendix  A. 

Summary  of  Recommendations.  We  recommend  that  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics: 

•  Enforce  the  requirement  that  program  managers  propose  and  milestone 
decision  authorities  establish  program-specific  exit  criteria  for  exiting 
acquisition  decision  points  for  major  Defense  acquisition  programs; 

•  Establish  exit  criteria  for  exiting  the  engineering  and  manufacturing 
development  phases  for  the  Multifunctional  Information  Distribution  System- 
Low  Volume  Terminal  and  the  VIRGINIA  class  submarine  programs;  and 

•  Revise  mandatory  instructions  for  the  Consolidated  Acquisition  Reporting 
System  to  require  program  managers  to  report  on  the  status  toward  satisfying 
exit  criteria  requirements  in  the  Defense  Acquisition  Executive  Summary. 

We  also  recommend  that  the  Deputy  for  Systems  Acquisition,  U.S.  Army  Tank- 
automotive  and  Armaments  Command,  establish  exit  criteria  for  the  Wide  Area 
Munition. 

Management  Comments.  We  provided  a  draft  of  this  report  on  October  11,  2()00. 

We  did  not  receive  comments  from  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics.  We  received  comments  from  the  Deputy  for  Systems 
Acquisition,  U.S.  Army  Tank-automotive  and  Armaments  Command,  who  agreed  with 
our  recommendation  applicable  to  the  Wide  Area  Munition,  and  the  (Ilommander,  Space 
and  Naval  Warfare  Systems  Command,  who  did  not  concur  with  the  finding  applicable 
to  the  Multifunctional  Information  Distribution  System-Low  Volume  Terminal  but 
agreed  to  implement  the  related  recommendation.  A  discussion  of  the  management 
comments  is  in  the  Findings  section  of  the  report,  and  the  complete  text  is  in  the 
Management  Comments  section. 

Audit  Response.  The  Army  and  the  Navy  comments  were  responsive  and  additional 
comments  are  not  required.  Because  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  did  not  respond  to  the  draft  report,  we  request  that  he 
provide  comments  on  this  final  report  by  February  12,  2001 . 
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Background 


This  report  discusses  the  extent  that  milestone  decision  authorities  used  exit 
criteria  as  a  management  tool  for  determining  program  manager’s  progress  in 
achieving  final  program  goals.  DoD  Regulation  5000,2-R,  “Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs,”  May  11,  1999, 
require  milestone  decision  authorities  to  use  exit  criteria  as  one  of  the  tools 
available  for  use  in  deciding  whether  an  acquisition  program  should  progress 
within  an  acquisition  phase  or  continue  to  the  next  phase  at  milestone  decision 
points.  At  Q^estone  decision  points,  program  managers  are  to  propose  the  exit 
criteria  for  the  next  phase  of  the  program,  and  milestone  decision  authorities  are 
to  review,  revise  as  needed,  and  document  the  approved  exit  criteria  in  their 
acquisition  decision  memorandums.  As  a  set  of  internally  developed 
benchmarks,  die  exit  criteria  proposed  by  program  managers  represent  the 
issues  and  factors  that  will  indicate  that  Ae  program  is  satisfactorily  progressing 
toward  final  program  goals.  The  exit  criteria  are  not  to  repeat  the  minimum 
required  accomplishments  for  each  acquisition  phase  contained  in  the  approved 
program  baseline  and  DoD  Regulation  5000.2-R. 

DoD  Regulation  5000.2-R  identifies  the  milestone  decision  points  and  the  timing 
of  exit  criteria  approval  as  follows: 


Milestones 


Phase  0  Concept 
Exploration 

Phase  I 

Program  Definition 
And  Risk  Reduction 

Phase  11  Engineering 
and  Manufacturing 
Development 

1 

r  1 

r  ^ 

Establish  exit 

1 

j  Establish  exit  criteria 

Entry  into 

criteria  for 

1  for  low-rate  initial 

fiill-rate 

Phase  I 

1  production  and  Phase  II 

production 

In  developing  exit  criteria,  program  managers  are  to  identify  criteria  that  will 
demonstrate  to  the  milestone  decision  authority  that  the  projgram  is  satisfactorily 
progressing  toward  achieving  final  program  goals.  Objective  exit  criteria, 
which  lead  to  satisfying  final  program  goals,  can  serve  as  an  invaluable  tool  to 
the  milestone  decision  authorities  for  measuring  interim  as  well  as  long-term 
program  progress.  Conversely,  exit  criteria  that  duplicate  existing  requirements 
contribute  little  to  the  milestone  decision  authorities’  ability  to  decide  whether 
an  acquisition  program  should  progress  within  an  acquisition  phase  or  continue 
to  the  next  phase  at  milestone  decision  points. 
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For  management  purposes,  DoD  Regulation  5000.2-R  requires  program 
managers  for  major  Defense  acquisition  programs  to  report  the  stams  toward 
satisfying  exit  criteria  requirements  in  the  quarterly  Defense  Acquisition 
Executive  Summary  (DAES).  This  reporting  requirement  enables  milestone 
decision  authorities  to  quickly  react  to  potential  problem  areas  between 
milestone  decision  points.  Appendix  B  provides  definitions  of  technical  terms 
used  in  the  report. 

Public  Law.  DoD  Regulation  5000.2-R  links  the  requirement  for  exit  criteria 
to  section  2220,  title  10,  United  States  Code  (10  U.S.C.  2220),  “Performance 
Based  Management:  Acquisition  Programs.”  In  addition,  10  U.S.C.  2220 
requires  that  the  Secretary  of  Defense  define  cost,  performance,  and  schedule 
goals  for  major  Defense  acquisition  programs  for  each  phase  of  the  acquisition 
process. 

Programs  Selected  for  Audit.  To  perform  the  audit,  we  selected  the  following 
nine  major  Defense  programs  for  review  that  the  program  offices  estimated  to 
total  $78  billion  in  development  and  procurement  costs: 

•  the  Army’s  Wide  Area  Munition,  All  Source  Analysis  System,  and 
Multiple  Launch  Rocket  System  Upgrade', 

•  the  Navy’s  Multifunctional  Information  Distribution  System-Low 
Volume  Terminal  (MIDS-LVT)',  Extended  Range  Guided  Munition,  and 
VIRGINIA  Class  Submarine';  and 

•  the  Air  Force’s  Wind  Corrected  Munition  Dispenser,  National  Airspace 
System',  and  Advanced  Extremely  High  Frequency  Program'. 

The  Under  Secreta^  of  Defense  for  Acquisition,  Technology,  and  Logistics  was 
the  milestone  decision  authority  for  three  of  the  five  major  Defense  acquisition 
programs  selected  for  our  review.  The  authority  for  making  milestone  decisions 
for  the  remaining  two  major  Defense  acquisition  programs  were  the  Component 
Acquisition  Executives  or  their  designees.  For  the  other  four  programs  (two 
Army,  one  Navy,  and  one  Air  Force),  the  milestone  decision  authority  was  the 
Component  Acquisition  Executive  or  their  designee.  Appendix  C  provides 
program  information  on  the  nine  programs  reviewed. 

Objectives 


The  audit  objective  was  to  evaluate  milestone  decision  authorities’  use  of  exit 
criteria  to  track  program  progress  in  important  technical,  schedule,  and 
management  risk  areas.  Specifically,  we  determined  whether  milestone  decision 
authorities  were  using  appropriate  exit  criteria  to  track  acquisition  program 


‘  Major  Defense  acquisition  program. 


progress  in  meeting  program  goals  and  to  aid  in  deciding  whether  programs 
should  progress  within  an  acquisition  phase  or  continue  to  the  next  acquisition 
phase.  In  addition,  we  evaluated  the  management  control  program  as  it  relates 
to  our  audit  objective.  Appendix  A  discusses  the  scope  and  methodology  as 
well  as  the  management  control  program. 
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A.  Establishing  Program-Specific  Exit 
Criteria 

For  seven  of  the  nine  programs  reviewed,  milestone  decision  authorities 
did  not  ensure  that  the  program  managers  proposed  program-specific  exit 
criteria  for  use  at  future  milestone  decision  point(s).  This  condition 
occurred  because: 

•  the  milestone  decision  authorities  for  four  programs  did  not  enforce 
the  requirement  that  program  managers  propose  exit  criteria  for  use 
at  future  milestone  decision  point(s),  and 

•  program  managers  for  seven  of  the  nine  programs  proposed  one  or 
more  exit  criteria  that  repeated  the  minimum  requirements  for 
acquisition  milestone  phases  contained  in  the  approved  program 
baseline  and  DoD  Regulation  5000.2-R. 

As  a  result,  the  milestone  decision  authorities  were  limited  in  their 
ability  to  use  exit  criteria  as  a  management  tool  to  determine  whether 
programs  under  their  review  and  oversight  should  progress  within  an 
acquisition  phase  or  continue  into  the  next  acquisition  phase  at  milestone 
decision  points. 

Exit  Criteria  Policy 

Defense  Policy.  The  requirement  for  establishing  exit  criteria  as  a  management 
tool  for  use  in  the  acquisition  process  was  initially  included  in  the  February  23, 
1991,  version  of  DoD  Instruction  5000.2,  “Defense  Acquisition  Managenient 
Policies  and  Procedures.  ”  The  Instruction  required  that  the  milestone  decision 
authority,  at  milestone  decision  points,  establish  program-specific  results,  or 
exit  criteria,  for  the  next  phase(s)  of  the  acquisition  process  for  Defense 
acquisition  programs.  The  Instruction  defined  exit  criteria  as  critical  results  that 
must  be  attained  during  the  next  acquisition  phase,  and  stated  that  failure  to 
meet  the  exit  criteria  would  halt  the  progress  of  a  system  towards  the  next 
milestone  decision  point.  Furthermore,  the  Instruction  required  that  exit  criteria 
be  program-specific  accomplishments  that  did  not  repeat  the  minimum  required 
accomplishments  for  each  acquisition  phase  contained  in  the  Instruction  and  in 
the  acquisition  program  baseline. 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology,^  incorporated 
the  exit  criteria  requirements  of  DoD  Instruction  5000.2  in  DoD  Regulation 
5000.2-R,  March  15,  1996.  For  major  Defense  acquisition  programs,  the 
Regulation  requires  that  the  milestone  decision  authority  establish  exit  criteria 
for  the  program  definition  and  risk  reduction  phase  (phase  I)  and  the  engineering 
and  manufacturing  development  phase  (phase  11)  of  the  acquisition  process. 


^  Renamed  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  in  October  1999. 
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The  Regulation  requires  that  the  milestone  decision  authority  document  the  exit 
criteria  in  the  acquisition  decision  memorandum  before  the  program  enters  each 
acquisition  phase.  Change  4  to  DoD  Regulation  5(X)0.2-R,  May  11,  1999, 
added  the  requirement  that  the  milestone  decision  authority  establish  exit  criteria 
for  the  low-rate  initial  production  decision  at  the  engineering  and  manufacturing 
development  decision  point.  In  addition,  the  Regulation  states  that  the 
requirements  in  the  Regulation  are  to  serve  as  a  model  for  managing  other  than 
major  Defense  acquisition  programs. 

Military  Departments’  Policy.  The  Military  Departments  extended  the 
requirement  for  using  exit  criteria  as  defined  in  DoD  Regulation  5000.2-R  to  all 
acquisition  programs  in: 

•  Army  Regulation  70-1,  “Research,  Development,  and  Acquisition;  Army 
Acquisition  Policy,”  January  15,  1998; 

•  Secretary  of  the  Navy  Instruction  5000. 2B,  “Implementation  of 
Mandatory  Procedures  for  Major  and  Non-Major  Defense  Acquisition 
Programs  and  Major  and  Non-Major  Information  Technology 
Acquisition  Programs,”  December  6,  1996;  and 

•  Air  Force  Policy  Directive  63-1,  “Acquisition  System,”  August  31, 

1993. 

Establishing  Exit  Criteria  on  the  Nine  Programs  Reviewed 


All  acquisition  programs  do  not  pass  through  each  milestone  decision  point 
where  DoD  Regulation  5000.2-R  requires  the  milestone  decision  authority  to 
establish  exit  criteria.  Some  programs,  such  as  the  Army’s  All  Source  Analysis 
System  and  Multiple  Launch  Rocket  System  Upgrade  and  the  Navy’s  Extended 
I^ge  Guided  Munition  and  MIDS-LVT,  combined  the  program  definition  and 
risk  reduction  phase  with  the  engineering  and  manufacturing  development 
phase.  For  those  programs,  separate  exit  criteria  were  not  necessary  for  the 
program  definition  and  risk  reduction  phase.  Similarly,  the  Army’s  All  Source 
Analysis  System  and  the  Air  Force’s  Advanced  Extremely  High  Frequency 
Program  did  not  include  a  low-rate  initial  production  decision  in  their 
acquisition  strategies.  Consequently,  exit  criteria  for  low-rate  initial  production 
did  not  apply  to  those  programs. 

Accordingly,  for  the  nine  programs  reviewed,  we  identified  23  decision  points 
for  which  milestone  decision  authorities  needed  to  establish  exit  criteria.  Four 
of  the  decision  points  covered  the  Wide  Area  Munition  product  improvement 
programs.  Although  the  milestone  decision  authority  established  exit  criteria  for 
the  product  improvement  programs,  he  did  not  define  whether  the  exit  criteria 
were  for  the  low-rate  initial  production  decision  or  for  exiting  the  engineering 
and  manufacturing  development  phase  of  the  acquisition  process.  In  addition,  as 
shown  in  the  following  table,  the  milestone  decision  authorities  did  not  establish 
exit  criteria  for  4  of  the  23  decision  points. 


5 


The  Regulation  requires  that  the  milestone  decision  authority  document  the  exit 
criteria  in  the  acquisition  decision  memorandum  before  the  program  enters  each 
acquisition  phase.  Change  4  to  DoD  Regulation  5000. 2-R,  May  11,  1999, 
added  the  requirement  that  the  milestone  decision  authority  establish  exit  criteria 
for  the  low-rate  initial  production  decision  at  the  engineering  and  manufacturing 
development  decision  point.  In  addition,  the  Regulation  states  that  the 
requirements  in  the  Regulation  are  to  serve  as  a  model  for  managing  other  than 
major  Defense  acquisition  programs. 

MUitary  Departments’  Policy.  The  Military  Departments  extended  the 
requirement  for  using  exit  criteria  as  defined  in  DoD  Regulation  5000. 2-R  to  all 
acquisition  programs  in: 

•  Army  Regulation  70-1,  “Research,  Development,  and  Acquisition;  Army 
Acquisition  Policy,”  January  15,  1998; 

•  Secretary  of  the  Navy  Instruction  5000. 2B,  “Implementation  of 
Mandatory  Procedures  for  Major  and  Non-Major  Defense  Acquisition 
Programs  and  Major  and  Non-Major  Information  Technology 
Acquisition  Programs,”  December  6,  1996;  and 

•  Air  Force  Policy  Directive  63-1,  “Acquisition  System,”  August  31, 

1993. 

Establishing  Exit  Criteria  on  the  Nine  Programs  Reviewed 

All  acquisition  programs  do  not  pass  through  each  milestone  decision  point 
where  DoD  Regulation  5000. 2-R  requires  the  milestone  decision  authority  to 
establish  exit  criteria.  Some  programs,  such  as  the  Army’s  All  Source  Analysis 
System  and  Multiple  Launch  Rocket  System  Upgrade  and  the  Navy’s  Extended 
Range  Guided  Munition  and  MIDS-LVT,  combined  the  program  definition  and 
risk  reduction  phase  with  the  engineering  and  manufacturing  development 
phase.  For  those  programs,  separate  exit  criteria  were  not  necessary  for  the 
program  definition  and  risk  reduction  phase.  Similarly,  the  Army’s  All  Source 
Analysis  System  and  the  Air  Force’s  Advanced  Extremely  High  Frequency 
Program  did  not  include  a  low-rate  initial  production  decision  in  their 
acquisition  strategies.  Consequently,  exit  criteria  for  low-rate  initial  production 
did  not  apply  to  Aose  programs. 

Accordingly,  for  the  nine  programs  reviewed,  we  identified  23  decision  points 
for  which  milestone  decision  authorities  needed  to  establish  exit  criteria.  Four 
of  the  decision  points  covered  the  Wide  Area  Munition  product  improvement 
programs.  Although  the  milestone  decision  authority  established  exit  criteria  for 
the  product  improvement  programs,  he  did  not  define  whether  the  exit  criteria 
were  for  the  low-rate  initial  production  decision  or  for  exiting  the  engineering 
and  manufacmring  development  phase  of  the  acquisition  process.  In  addition,  as 
shown  in  the  following  table,  the  milestone  decision  authorities  did  not  establish 
exit  criteria  for  4  of  the  23  decision  points. 
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AvailabUity  of  Exit  Criteria  for  Future  Program  Reviews 

Program 

Engineering  and 

Definition  and 

Low-Rate  Initial 

Manufacturing 

Risk  Reduction 

Production 

Development 

Program 

Phase 

Decision 

Phase 

Amy 

All  Source  Analysis  System 
Multiple  Launch  Rocket  System 

X 

-Guided  Rocket 

X 

X 

-Launcher 

X 

Wide  Area  Munition 

-Basic 

X 

Not  Established 

-Product  Improvement-l 

Undeterminable 

Undeterminable 

-Product  Improvement-2 

Undeterminable 

Undeterminable 

Navy 

Extended  Range  Guided  Munition 

X 

X 

MIDS-LVT 

X 

Not  Established 

VIRGINIA  Class  Submarine 

X 

X 

Not  Established 

Air  Force 

Advanced  Extremely  High 

X 

Frequency  Program 

National  Airspace  Program 

X 

Not  Established 

X 

Wind  Corrected  Munition 

X 

X 

Dispenser 

X-  The  milestone  decision  authority  establish  exit  criteria 

Undeterminable-  Exit  criteria  not  designated  for  a  specific  decision  point. 

Not  Establish-  Milestone  decision  authority  did  not  establish  exit  criteria  for  this  decision  point. 

Need  for  Exit  Criteria 

Milestone  decision  authorities  for  four  of  the  nine  programs  reviewed  did  not 
require  program  managers  to  propose  exit  criteria  for  use  at  all  applicable 
milestone  decision  points.  Tlus  condition  occurred  because  the  milestone 
decision  authorities  for  the  four  programs  (Wide  Area  Munition,  MIDS-LVT, 
VIRGINIA  class  submarine  and  the  National  Airspace  Program)  did  not  enforce 
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the  requirement  that  program  managers  propose  exit  criteria  for  use  at  future 
milestone  decision  points.  In  the  following  paragraphs,  we  discuss  the  three 
programs  that  require  exit  criteria  for  upcoming  decisions. 

Wide  Area  Munition.  The  Wide  Area  Munition  consists  of  a  basic  program 
and  two  product  improvement  programs.  For  the  basic  program,  the  Prograni 
Executive  Officer  for  Armored  Systems  Modernization^  established  exit  criteria 
for  low-rate  initial  production,  but  not  for  the  engineering  and  manufacturing 
development  phase.  The  milestone  decision  authority  established  the  low-rate 
initial  production  exit  criteria  for  the  basic  program  at  the  same  time  that  he 
made  the  low-rate  initial  production  decision  on  June  18,  1996.  However,  the 
milestone  decision  authority  did  not  require  the  program  manager  to  propose 
exit  criteria  for  the  engineering  and  manufacturing  development  phase.  When 
asked  why  not,  the  project  officer  stated  that  the  exit  criteria  established  for  this 
decision  were  applicable  for  the  low-rate  initial  production  decision  and  the 
engineering  and  manufacturing  development  phase.  However,  the  project 
officer  was  not  able  to  provide  documentation  to  support  that  assertion.  For  the 
two  product  improvements,  the  milestone  decision  authority  did  not  identify 
whether  the  exit  criteria  established  at  the  engineering  and  manufacturing 
development  decision  points  were  applicable  to  the  low-rate  initial  production 
decision  or  for  the  engineering  and  manufacturing  development  phase  of  the 
acquisition  process. 

More  recently,  the  Wide  Area  Munition  program  office  informed  us  that  it  is 
restructuring  the  program.  The  restructured  program  will  be  called  the 
Advanced  Hornet  and  will  have  modified  exit  criteria.  The  user  is  in  the 
process  of  revising  the  operational  requirements  document  for  the  restructured 
program. 

MIDS-LVT.  For  the  MIDS-LVT  program,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  approved  entry  into  the  engineering  and 
manufacturing  development  phase  of  the  acquisition  process  on  December  17, 
1993.  However,  he  did  not  approve  the  exit  criteria  that  the  program  manager 
proposed  for  exiting  that  phase.  Instead,  he  shifted  the  focus  of  Ae  program 
and  instructed  the  program  manager  to  submit  updated  exit  criteria.  In 
response,  the  program  manager  proposed  updated  criteria  for  the  engineering 
and  manufacturing  development  phase  on  October  5,  1994.  On  review  of  the 
proposed  criteria,  the  milestone  decision  authority  established  the  criteria  for  the 
low-rate  initial  production  decision,  but  not  the  exit  criteria  for  the  engineering 
and  manufacturing  development  phase.  The  program  manager  believed  that  the 
milestone  decision  authority  would  establish  those  exit  criteria  at  the  program 
review  for  the  low-rate  initial  production  decision  in  May  2000.  However,  the 
milestone  decision  authority  did  not  require  the  program  manager  to  propose 
exit  criteria  for  the  engineering  and  manufacturing  development  phase  at  the 
program  review  held  on  May  11,  2000. 


^  On  June  23,  1997,  the  milestone  decision  authority  was  delegated  to  the  Deputy  for  Systems 
Acquisition,  U.S.  Army  Tank-automotive  and  Armaments  Command. 
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VIRGINIA  Class  Submarine.  For  the  VIRGINIA  class  subinarine,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  approved  entry 
into  the  engineering  and  manufacturing  development  phase  of  the  acquisition 
process  on  June  30,  1995.  At  that  time,  the  milestone  decision  authority 
established  exit  criteria  for  the  low-rate  initial  production  decision.  However, 
the  milestone  decision  authority  did  not  require  the  program  manager  to  propose 
exit  criteria  for  the  engineering  and  manufacturing  development  phase.  When 
asked,  the  program  manager  stated  that  he  did  not  believe  that  he  could  develop 
meaningful  exit  criteria  for  a  decision  that  was  12  years  in  the  future.  He 
further  stated  that  the  program  office  plaimed  to  propose  exit  criteria  for 
engineering  and  manufacturing  development  in  FY  2002,  as  part  of  the 
overarching  integrated  product  team  review. 

Requirement  for  Exit  Criteria.  Milestone  decision  authorities  should  have 
required  the  program  managers  to  propose  different  and  more  rigorous  criteria 
for  exiting  the  engineering  and  manufacturing  development  phase  for  the  three 
acquisition  programs.  As  required  by  DoD  Regulation  5000. 2-R,  program 
managers  must  demonstrate  more  rigorous  cost,  schedule,  and  performance 
criteria  between  the  low-rate  initial  production  decision  and  the  full-rate 
production  milestone  decision  points.  For  example,  the  Regulation  requires  the 
program  manager  to  demonstrate,  through  testing,  that  the  system  is  potentially 
operationally  effective  and  suitable  before  the  low-rate  initial  production 
decision.  Before  the  fiill-rate  production  decision,  the  Regulation  requires  the 
program  manager  to  demonstrate  that  the  system  is  operationally  effective  and 
suitable.  Consequently,  without  criteria,  the  milestone  decision  authority  and 
the  program  manager  do  not  have  the  benefit  of  a  focused  approach  that  exit 
criteria  bring  to  the  decision-making  process. 

Establishing  Program-Specific  Requirements  for  Exit  Criteria 

DoD  Regulation  5000.2-R  indicates  that  exit  criteria  should  be  a  level  of 
demonstrated  performance  (for  example,  a  level  of  engine  thrust),  the 
accomplishment  of  a  process  at  a  certain  level  of  efficiency  (for  example, 
manufacturing  yield)  or  a  successful  accomplishment  of  an  event  (for  example, 
first  flight)  that  shows  that  the  program  is  progressing  satisfactorily  toward 
program  goals.  These  are  examples  of  program-specific  exit  criteria.  With  the 
exception  of  the  Advanced  Extremely  High  Frequency  Program  and  the 
Multiple  Launch  Rocket  System,  program  managers  for  the  nine  programs 
reviewed  proposed  one  or  more  exit  criteria  that  were  not  program  specific  as 
required  by  the  Regulation,  and  instead  repeated  the  minimum  required  program 
accomplishments  contained  in  the  approved  program  baseline  or  DoD 
Regulation  5000.2-R.  Appendix  D  lists  the  exit  criteria  established  for  the  nine 
programs  reviewed  and  identifies  the  exit  criteria  that  were  not  program 
specific. 

The  following  paragraphs  discuss  examples  of  program-specific  exit  criteria  and 
nonprogram-specific  exit  criteria  that  program  managers  proposed  and  milestone 
decision  authorities  approved. 
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Program-Specific  Exit  Criteria.  Program  managers  did  propose  exit  criteria 
that  were  program  specific.  For  example,  exit  criteria  established  for  the  Wind 
Corrected  Munition  Dispenser  included  a  requirement  that  the  program  manager 
demonstrate  dispenser  operation  in  medium-to-high  altitudes  before  the  program 
could  enter  low-rate  initial  production.  Similarly,  exit  criteria  for  the 
VIRGINIA  class  submarine  included  a  requirement  that  the  program  manager 
provide  an  estimate  of  the  submarine’s  inboard  electronics  configuration  before 
the  program  definition  and  risk  reduction  phase  could  be  exited.  On  the 
Extended  Range  Guided  Munition  program,  exit  criteria  required  that  the 
program  manager  demonstrate  a  successful  target  engagement  of  an  extended 
range  guided  munition  that  was  land-based  and  gun-fired  before  the  milestone 
decision  authority  granted  approval  to  enter  low-rate  initial  production. 
Satisfaction  of  these  program-specific  exit  criteria  will  demonstrate  to  the 
milestone  decision  authorities  whether  the  programs  are  progressing 
satisfactorily  toward  program  goals. 

Exit  Criteria  That  are  Not  Program  Specific.  One  or  more  exit  criteria  for 
seven  of  the  nine  programs  reviewed  were  not  program  specific.  On  the  Wide 
Area  Munition  program,  for  example,  exit  criteria  required  the  program 
manager  to  demonstrate  the  achievement  of  performance  requirements  specified 
in  the  required  operational  capability  document.  Similarly,  on  the  Wind 
Corrected  Munition  Dispenser  program,  exit  criteria  required  the  progr^ 
manager  to  submit  documentation  to  support  the  decision  to  exit  the  engineering 
and  manufacturing  development  phase.  Such  nonspecific  exit  criteria  do  not 
provide  milestone  decision  authorities  with  additional  information  to  make 
programmatic  decisions. 

Examples  Identified  in  Earlier  Audit  Reports.  In  1996,  1997,  and  2000,  we 
reported  on  other  acquisition  programs  that  did  not  use  established  exit  criteria 
for  making  milestone  decisions.  In  Inspector  General,  DoD,  Report 
No.  97-018,  “The  Patriot  Advanced  Capabilities-3  Program,”  November  6, 
1996,  we  reported  that  the  exit  criteria  for  full-rate  production  did  not  require 
that  missile  reliability,  a  critical  performance  parameter,  be  stabilized  before 
production.  The  Office  of  the  Secretary  of  Defense  (OSD)  revised  the  exit 
criteria  to  include  achievement  of  proper  missile  reliability  before  entering  full- 
rate  production.  In  Inspector  General,  DoD,  Report  No.  97-199,  “The 
Minuteman  III  Guidance  Replacement  Program,”  My  29,  1997,  we  reported 
that  the  exit  criteria  established  for  the  low-rate  initial  production  were  too 
general  in  nature.  On  March  6,  1997,  the  Program  Executive  Office  for  Space 
Programs  approved  revised  exit  criteria,  which  listed  more  specific 
accomplishments  as  prerequisites  before  the  program  manager  could  award  the 
low-rate  initial  production  contract.  There  was  no  indication  that  the  OSD  took 
measures  to  ensure  that  other  programs  did  not  have  the  same  deficiency. 

Effective  Use  of  Exit  Criteria 


Milestone  decision  authorities  for  the  Wind  Corrected  Munition  Dispenser,  the 
National  Airspace  System,  and  the  Multiple  Laimch  Rocket  System  Upgrade 
used  the  established  exit  criteria  as  an  integral  part  in  managing  their  programs. 
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Wind  Corrected  Munition  Dispenser.  As  evidenced  in  the  acquisition  decision 
memorandum,  the  Principal  Deputy  Assistant  Secretary  of  the  Air  Force  for 
Acquisition  and  Management  used  the  exit  criteria  as  an  integral  part  of  the 
decision-making  process.  In  the  August  3,  1998,  memorandum,  the  milestone 
decision  authority  identified  the  established  low-rate  initial  production  exit 
criteria  and  explained  how  the  program  manager  had  demonstrated  achievement 
of  the  exit  criteria.  Clearly,  this  milestone  decision  authority  used  the  exit 
criteria  as  a  management  tool  in  the  decision-making  process. 

National  Airspace  System.  For  the  National  Airspace  System  program,  the 
Principal  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition  and 
Management  requested  and  received  documentation  from  the  program  manager 
that  demonstrated  that  exit  criteria  established  for  the  program  definition  and 
risk  reduction  and  engineering  and  manufacturing  development  phases  were 
met. 

Multiple  Launch  Rocket  System  Upgrade.  An  integrated  product  team, 
representing  the  Army  Program  Executive  Officer  for  Tactical  Missiles,  used 
the  exit  criteria  for  monitoring  the  program  manager’s  progress  in  satisfying 
requirements  for  entry  into  low-rate  initial  production.  This  review  occurred 
1  year  before  the  low-rate  initial  production  decision  point.  As  a  result  of  the 
review,  the  program  manager  was  able  to  provide  the  mtegrated  product  team 
with  additional  information  demonstrating  the  satisfaction  of  the  exit  criteria 
before  the  low-rate  initial  production  decision  point. 

Summary 


Milestone  decision  authorities  were  not  enforcing  the  requirement  that  program 
managers  propose  program-specific  exit  criteria  for  use  as  a  management  tool  in 
reviewing  and  overseeing  acquisition  programs.  In  the  absence  of  progr^- 
specific  exit  criteria  for  major  acquisition  decision  points,  milestone  decision 
authorities  were  limited  in  their  ability  to  use  exit  criteria  as  a  management  tool 
to  determine  whether  programs  under  their  review  and  oversight  should 
progress  within  an  acquisition  process  or  continue  into  the  next  acquisition 
phase  at  milestone  decision  points. 

Management  Comments  on  the  Finding  and  Audit  Response 


Man^ement  Comments  on  the  MIDS-LVT  Exit  Criteria.  The  Commmder, 
Space  and  Naval  Warfare  Systems  Command,  nonconcurred  with  the  finding 
assertion  for  the  MIDS-LVT  program  that  four  of  the  exit  criteria  established 
for  entering  the  low-rate  initial  production  phase  of  the  acquisition  process  were 
not  program  specific.  He  stated  that  the  four  exit  criteria  (closing  out  all  critical 
design  review  issues,  completing  qualification  testing,  conducting  the  production 
readiness  review,  and  conducting  a  physical  configuration  audit)  were  program 
specific  and  provided  the  milestone  decision  authority  with  information  to  make 
the  appropriate  programmatic  decision.  The  complete  text  is  in  the  Management 
Conunents  section  of  this  report. 
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Audit  Response.  We  agree  that  the  four  exit  criteria  in  question  will  provide 
the  milestone  decision  authority  with  information  to  make  the  appropriate 
programmatic  decision.  As  stated  in  the  finding,  DoD  Regulation  5000.2 
requires  that  the  exit  criteria  be  program  specific  and  not  repeat  the  minimum 
required  accomplishments  contained  in  the  approved  program  baseline  or  DoD 
Regulation  5000.2.  As  a  result  of  the  conunents  received,  we  reviewed  our 
assessment  and  determined  that  DoD  Regulation  5000. 2-R  does  not  require  a 
production  readiness  review;  therefore,  we  adjusted  Appendix  D  accordingly. 
However,  the  remaining  three  exit  criteria  in  question  were  not  program  specific 
because  they  repeated  required  accomplishments  contained  in  DoD  Regulation 
5000.2. 

Recommendations,  Management  Comments,  and  Audit 
Response 

A.l.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics: 

a.  Enforce  the  requirement  that  program  managers^  propose  and^ 
milestone  decision  authorities  establish  program-specific  criteria  for  exiting 
acquisition  decision  points  for  major  Defense  acquisition  programs; 

b.  Establish  exit  criteria  for  exiting  the  engineering  and 
manufacturing  development  phases  for  the  Multifunctional  Information 
Distribution  System-Low  Volume  Terminal  and  the  VIRGINIA  class 
submarine  programs. 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  did  not  comment  on  the  recommendation.  We 
request  that  he  provide  comments  in  response  to  the  final  report. 

Although  not  required  to  comment,  the  Commander,  Space  and  Naval  Warfare 
Systems  Command,  stated  that  exit  criteria  for  the  MIDS-LVT  full-rate 
production  decision  were  being  developed  and  would  be  submitted  to  the 
milestone  decision  authority  for  approval. 

A.2.  We  recommend  that  the  Deputy  for  Systems  Acquisition,  U.S.  Army 
Tank-automotive  and  Armaments  Command,  establish  exit  criteria  for  the 
low-rate  initial  production  decision  and  the  engineering  and  manufacturing 
development  phase  for  the  product  improvement  programs-1  and  -2  of  the 
Wide  Area  Munition. 

Management  Comments.  The  Deputy  for  Systems  Acquisition,  U.S.  Army 
Tank-automotive  and  Armaments  Command,  concurred  and  estimated  that 
revision  of  the  exit  criteria  will  be  completed  by  January  19,  2001. 
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B.  Exit  Criteria  Status  Reporting 

Program  managers  for  three  of  the  five  major  Defense  acquisition 
programs  reviewed  did  not  report  their  status  toward  atoining  exit 
criteria  requirements  in  the  quarterly  Defense  Acquisition  Executive 
Summary.  Program  managers  did  not  report  exit  criteria  status  because 
two  program  managers  did  not  understand  the  exit  criteria  reporting 
requirements  established  in  DoD  Regulation  5000.2-R,  and  one  program 
manager  did  not  have  exit  criteria  established  to  report  against. 
Moreover,  the  Defense  Acquisition  Executive  did  not  have  procedures  in 
place  to  enforce  the  reporting  requirement.  As  a  result,  nailestone 
decision  authorities  and  OSD  action  officers  did  not  have  information  to 
use  as  a  management  tool  for  assessing  each  program’s  progress  toward 
satisfying  exit  criteria  requirements  and  for  providing  direction,  when 
needed,  between  milestone  decision  points. 

Exit  Criteria  Status  Reporting  Policy 


DoD  Regulation  5000.2-R  requires  program  managers  for  niajor  Defense 
acquisition  programs  to  report  the  status  toward  attaining  exit  criteria 
requirements  in  the  quarterly  Defense  Acquisition  Executive  Sunimary  (DAES). 
Program  managers  are  to  prepare  the  quarterly  DAES  using  the  Consolidated 
Acquisition  Reporting  System  (CARS).  The  CARS  is  a  personal-computer- 
based,  entry  and  reporting  software  package  that  maintains  and  reports 
information  on  major  Defense  acquisition  programs.  The  CARS  has  mandatory 
instructions  on  how  to  prepare  and  review  the  quarterly  DAES  and  defines 
where  in  the  DAES  the  program  manager  must  report  the  program’s  stams 
toward  achieving  exit  criteria  requirements. 

Reporting  Program  Status 


Program  managers  for  three  of  the  five  major  Defense  acquisition  programs 
reviewed  did  not  report  their  stams  toward  achieving  exit  criteria  requirements 
in  the  quarterly  DAES.  Program  managers  for  the  Advanced  Extremely  High 
Frequency  Program  and  the  Multiple  Launch  Rocket  System  did  not  understand 
the  exit  criteria  reporting  requirements  in  DoD  Regulation  5000.2-R  Md  the 
program  manager  for  the  VIRGINIA  class  submarine  did  not  have  exit  criteria 
established  to  report  against.  This  occurred  because  the  Defense  Acquisition 
Executive  did  not  have  procedures  in  place  through  the  mandatory  CARS 
instructions  to  enforce  the  reporting  requirement.  Although  the  CARS 
instructions  defines  the  process  that  OSD  action  officers  use  to  assess  and 
review  the  DAES  submission,  it  does  not  assign  the  OSD  action  officers  with 
responsibility  for  ensuring  that  program  managers  include  exit  criteria  stams 
information  in  the  DAES.  Program  managers,  when  asked,  gave  different 
reasons  for  not  reporting  exit  criteria  stams  information  in  the  DAES. 

Advanced  Extremely  High  Frequency  Program.  The  program  manager  for 
the  Advanced  Extremely  High  Frequency  Program  stated  that  she  was  not  aware 
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of  the  requirement  to  report  on  the  status  toward  achieving  exit  criteria 
requirements  in  the  DAES.  Moreover,  the  program  manager  stated  that  OSD 
action  officers  did  not  note  the  omission  during  their  review  of  the  quarterly 
DAES  submission.  As  a  result  of  our  review,  Ae  program  manager  began 
reporting  on  the  status  toward  achieving  exit  criteria  requirements  in  the  June 
2000  DAES.  Current  information  concerning  the  ability  of  a  program  manager 
to  achieve  established  exit  criteria  is  essential  in  that  it  provides  the  milestone 
decision  authority  with  the  opportunity  to  intercede  if  progress  is  not  deemed 
satisfactory. 

Multiple  Launch  Rocket  System  Upgrade.  The  program  manager  stoted  that 
he  reported  on  the  status  toward  achieving  the  program’s  exit  criteria  in  the 
DAES  on  an  exception  basis.  Specifically,  the  program  manager  explained  that 
his  DAES  submissions  on  exit  criteria  status  were  omitted  unless  he  was 
experiencing  problems  in  achieving  exit  criteria  requirements.  He  also  indicated 
that  OSD  action  officers  had  not  brought  the  exit  criteria  reporting  omission  to 
his  attention  for  corrective  action. 

VIRGINIA  Class  Submarine.  The  program  manager  for  the  VIRGINIA  class 
submarine  also  did  not  report  on  the  status  toward  attaining  the  program’s  exit 
criteria  in  the  DAES  because  he  did  not  propose,  and  the  milestone  decision 
authority  did  not  establish,  exit  criteria  for  the  engineering  and  manufacturing 
development  phase  of  the  acquisition  process. 

Condition  Previously  Reported.  In  Inspector  General,  DoD,  Report 
No.  99-075,  “Acquisition  of  the  SH-60R  Light  Airborne  Multipurpose  System 
Mark  III  Block  H  Upgrade,”  February  2,  1999,  we  also  reported  that  OSD 
action  officers  did  not  question  the  absence  of  exit  criteria  status  information  in 
the  DAES.  Based  on  the  report,  the  program  manager  agreed  to  begin  reporting 
on  the  program’s  exit  criteria  status  in  the  DAES.  There  was  no  indication  that 
the  OSD  took  any  measures  to  ensure  that  other  programs  did  not  have  the  same 
deficiency. 

CARS  Procedures 


The  CARS  instruction  identified  nine  indicator  categories  for  OSD  action 
officers’  examination.  However,  the  nine  indicators  did  not  include  a 
requirement  for  OSD  action  officers  to  review  exit  criteria  reporting 
information.  An  OSD  action  officer  confirmed  that  their  primary  review  focus 
was  on  the  nine  indicator  categories.  Without  this  focus  in  the  CARS 
instruction,  OSD  action  officers  were  not  verifyinjg  whether  program  managers 
were  reporting  on  their  exit  criteria  status  as  required.  To  remedy  this 
condition,  the  CARS  instruction  could  be  revised  to  include  exit  criteria 
requirements  as  an  indicator  category  for  OSD  action  officer  examination  or  to 
insert  CARS  validation  checks  on  DAES  submissions  for  the  program  managers 
to  include  exit  criteria  status  information  in  the  DAES. 
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Summary 


One  of  the  primary  purposes  of  establishing  and  monitoring  exit  criteria  is  to 
provide  milestone  decision  authorities  with  information  for  assessing  a 
program’s  progress  toward  satisfying  final  program  goals;  a  periodic  stams 
reporting  mechanism  is  essential.  Without  such  a  reporting  mechanism, 
milestone  decision  authorities  and  OSD  action  officers  did  not  have  information 
to  use  as  a  management  tool  for  assessing  each  program’s  progress  toward 
satisfying  exit  criteria  requirements  and  for  providing  direction,  when  needed, 
between  milestone  decision  points. 

Recommendation  and  Management  Comments 


B.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  revise  the  Consolidated  Acquisition  Reporting 
System  instructions  to  include  a  requirement  to  use  exit  criteria  as  an 
indicator  category  for  Office  of  the  Secretary  of  Defense  action  officer 
examination  or  to  insert  a  Consolidated  Acquisition  Reporting  System 
validation  check  on  Defense  Acquisition  Executive  Summ^  submissions  for 
program  managers  to  include  exit  criteria  status  information  in  the  Defense 
Acquisition  Executive  Summary. 

Management  Comments.  The  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  did  not  comment  on  the  recommendation.  We 
request  that  he  provide  coimnents  in  response  to  the  final  report. 
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Appendix  A.  Audit  Process 

Scope 


We  conducted  this  audit  from  January  through  September  2000,  and  reviewed 
documentation  dated  from  November  1986  through  Au^st  2000.  We 
performed  the  audit  using  requirements  pertaining  to  exit  criteria  in 
DoD  Regulation  5000. 2-R,  “Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  (MDAPs)  and  Major  Automated  Information  Systems 
(MAIS)  Acquisition  Programs,”  Change  4,  May  11,  1999. 

In  accomplishing  the  objective,  we  selected  nine  major  Defense  programs  to 
review:  three  each  from  the  Army,  Navy  and  Air  Force.  We  subjectively 
selected  the  programs  to  provide  a  variety  of  weapon  t5q)es  and  a  cross-section 
of  DoD  acquisition  organizations,  while  considering  the  amount  of  prior  audit 
coverage.  We  did  not  include  Major  Automated  Information  Systems  in  our 
review  because  exit  criteria  for  those  programs  will  be  covered  in  separate 
upcoming  audits.  The  nine  major  Defense  programs  selected  were: 

•  the  Army’s  Wide  Area  Munition,  All  Source  Analysis  System,  and 
Multiple  Launch  Rocket  System  Upgrade; 

•  the  Navy’s  MIDS-LVT,  Extended  Range  Guided  Mimition,  and 
VIRGINIA  Class  Submarine;  and 

•  the  Air  Force’s  Wind  Corrected  Munition  Dispenser,  National 
Airspace  System,  and  Advanced  Extremely  High  Frequency  Program. 

DoD- Wide  Corporate  Level  Government  Performance  and  Results  Act 
Goals.  In  response  to  the  Government  Performance  and  Results  Act ,  the 
Secretary  of  Defense  annually  establishes  DoD-wide  corporate  level  goals, 
subordinate  performance  goals,  and  performance  measures.  This  report  pertains 
to  achievement  of  the  following  corporate  level  goal  and  subordinate 
performance  goal. 

•  FY  2001  DoD  Corporate  Level  Goal  2:  Prepare  now  for  an  uncertain 
future  by  pursuing  a  focused  modernization  effort  that  maintains  U.S. 
qualitative  superiority  in  key  warfighting  capabilities.  Transfonn  the 
force  by  exploiting  the  Revolution  in  Military  Affairs,  and  reengineer 
the  Department  to  achieve  a  21st  century  infrastructure. (01 -DoD-2) 

•  FY  2001  Subordinate  Performance  Goal  2.4:  Meet  combat  forces’ 
needs  smarter  and  faster,  with  products  and  services  that  work  better 
and  cost  less,  by  improving  the  efficiency  of  DoD’s  acquisition 
processes.  (Ol-DoD-2.4) 
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General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  report  provides  coverage 
of  the  Defense  Weapons  Systems  Acquisition  high-risk  area. 

Methodology 


We  conducted  this  program  audit  in  accordance  with  auditing  standards  issued 
by  the  Comptroller  General  of  the  United  States,  as  implemented  by  the 
Inspector  General,  DoD,  and  included  such  tests  of  management  controls  as  we 
deemed  necessary.  We  did  not  use  computer-processed  data  to  perform  this 
audit. 

To  accomplish  the  audit  objective,  we  took  the  following  steps: 

•  determined  whether  exit  criteria  were  in  place  for  milestone  decision 
points  and  other  programmatic  events,  such  as  low-rate  initial 
production  decisions; 

•  determined  whether  the  program  managers  periodically  reported  the 
status  on  attaining  the  exit  criteria  requirements  to  milestone  decision 
authorities  for  their  review  and  use;  and 

•  determined  whether  the  milestone  decision  authorities  were  using  the 
exit  criteria  as  a  program  management  tool. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  the  DoD.  Further  details  are  available  upon  request. 

Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26, 
1996,  and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program 
Procedures,”  August  28,  1996,  require  DoD  organizations  to  implement  a 
comprehensive  system  of  management  controls  drat  provides  reasonable 
assurance  that  programs  are  operating  as  intended  and  to  evaluate  the  adequacy 
of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  In  accordance 
with  DoD  Directive  5000.1,  “Defense  Acquisition,”  March  15,  1996,  and  DoD 
Regulation  5000.2-R,  acquisition  managers  are  to  use  program  cost,  schedule, 
and  performance  parameters  as  control  objectives  to  implement  the  requirements 
of  DoD  Directive  5010.38.  Accordingly,  we  limited  our  review  to  management 
controls  directly  related  to  the  use  of  exit  criteria  for  major  defense  systems. 

Adequacy  of  Management  Controls.  We  identified  a  material  management 
control  weakness  as  defined  by  DoD  Instruction  5010.40,  “Management  Control 
(MC)  Program  Procedures,”  August  28,  1996.  Management  controls  were  not 
adequate  for  ensuring  that  program  managers  for  major  Defense  acquisition 
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program  established  exit  criteria  and  reported  the  status  of  achieving  approved 
exit  criteria  requirements  in  the  quarterly  Defense  Acquisition  Executive 
Summary.  Recommendations  A.l.  and  B.,  if  implemented,  will  ensure  that  exit 
criteria  are  established  and  that  the  status  of  approved  exit  criteria  is  routinely 
reported  in  the  Defense  Acquisition  Executive  Summary. 

Adequacy  of  Management’s  Self-Evaluation.  The  Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  did  not 
perform  a  self-evaluation  to  identify  material  management  control  weaknesses. 

Prior  Coverage 

During  the  last  5  years,  the  Inspector  General,  DoD,  issued  four  reports  that 
addressed  exit  criteria. 

Inspector  General,  DoD,  Report  No.  D-2000-187,  “The  Low-Rate  Initial 
Production  Decision  for  the  Joint  Biological  Point  Detection  System,” 
September  11,  2000 

Inspector  General,  DoD,  Report  No.  99-075,  “Acquisition  of  the  SH-60R  Light 
Airborne  Multipurpose  System  Mark  III  Block  n  Upgrade,”  February  2,  1999 

Inspector  General,  DoD,  Report  No.  97-199,  “The  Minuteman  III  Guidance 
Replacement  Program,”  July  29,  1997 

Inspector  General,  DoD,  Report  No.  97-018,  “The  Patriot  Advanced 
Capabilities-3  Program,”  November  4,  1996 
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Appendix  B.  Terms  and  Definitions 


Acquisition  Category.  An  acquisition  category  is  an  attribute  of  an  acquisition 
program  that  determines  the  program’s  level  of  review,  decision  authority,  pd 
procedures.  The  acquisition  categories  consist  of  I,  major  Defense  acquisition 
programs;  lA,  major  automated  information  systems;  II,  less  than  major 
systems;  and  III,  all  other  acquisition  programs.  In  addition.  Acquisition 
Category  I  programs  have  two  subcategories:  Acquisition  Category  ID 
programs,  where  the  milestone  decision  authority  is  the  Under  Secretary 
Defense  for  Acquisition,  Technology,  and  Logistics,  Md  Acquisition  Category 
IC  programs,  where  the  milestone  decision  authority  is  the  Component 
Acquisition  Executive. 

Acquisition  Phase.  An  acquisition  phase  represents  all  the  tasks  and  activities 
needed  to  bring  a  program  to  the  next  major  milestone.  Phases  provide  a  logical 
means  of  progressively  translating  broadly  stated  mission  needs  into  well- 
defined,  system-specific  requirements  and,  ultimately,  into  operationally 
effective,  suitable,  and  survivable  systems. 

Component  Acquisition  Executive.  A  single  official  within  a  component  who 
is  responsible  for  all  acquisition  functions  within  that  component. 

Defense  Acquisition  Executive  Summa^.  The  DAES  is  the  principal  Defense 
Acquisition  Executive  reporting  mechanism  for  tracking  program  status  between 
program  milestone  events. 

Engineering  and  Manufacturing  Development.  Engineering  and 
manufacturing  development  is  the  third  phase  of  the  acquisition  process  where 
the  program  Mly  develops,  engineers,  designs,  fabricates,  tests,  and  evaluates 
the  systems  and  the  principal  items  necessary  for  its  support. 

Exit  Criteria.  Exit  criteria  are  program-specific  accomplishments  that  must  be 
satisfactorily  demonstrated  before  a  program  can  progress  further  in  the  current 
acquisition  phase  or  continue  to  the  next  acquisition  phase. 

Low-Rate  Initial  Production.  Low-rate  initial  production  is  the  production  of 
a  system  in  limited  quantities  to  provide  articles  for  additional  operational  test 
and  evaluation,  to  establish  an  initial  production  base,  and  to  pennit  an  orderly 
increase  in  the  production  rate  that  will  lead  to  full-rate  production  after 
successful  completion  of  operational  testing. 

Milestone.  A  milestone  is  the  point  where  the  milestone  decision  authority 
decides  whether  to  start  or  continue  an  acquisition  program  in  the  acquisition 
process. 

Milestone  Decision  Authority.  A  milestone  decision  authority  is  the  individual 
designated  in  accordance  with  criteria  established  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics  to  approve  entry  of  an 
acquisition  program  into  the  next  phase. 
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Appendix  C.  Information  on  Nine  Programs 

Reviewed 


Advanced  Extremely  High  Frequency  Program.  The  Advanced  Extremely 
High  Frequency  Program  is  an  Air  Force  Acquisition  Category  ID  major 
Defense  acquisition  program.  The  Advanced  Extremely  High  Frequency 
satellite  system  provides  survivable,  jam-resistant,  worldwide,  secure 
communications  for  the  strategic  and  tactical  warfighter.  The  program  manager 
estimates  a  total  acquisition  cost  of  $2.5  billion.  The  milestone  decision 
authority  for  the  program  is  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics. 

All  Source  Analysis  System.  The  All  Source  Analysis  System  is  an  Army 
Acquisition  Category  II  major  system.  It  is  a  mobile,  tactically  deployable, 
computer-assisted  intelligence  and  electronic  warfare  processing  system.  The 
system  is  employed  during  peacetime  and  wartime  operational  environments  and 
is  capable  of  continuous  24-hours-a-day,  7  days-a-week  operation  under  tactical 
conditions.  The  program  office  estimates  a  total  acquisition  cost  of  $1 .2  billion. 
The  milestone  decision  authority  for  the  program  is  the  Arniy  Program 
Executive  Officer  for  Command,  Control,  and  Communication  Systems. 

Extended  Range  Guided  Munition.  The  Extended  Range  Guided  Munition  is 
a  Navy  Acquisition  Category  II  major  system.  The  Extended  Range  Guided 
Munition  projectile  is  rocket-assisted,  with  a  range  of  63  nautical  miles.  The 
Extended  Range  Guided  Munition  is  designed  to  provide  supporting  fire  for 
amphibious  assaults,  raids,  demonstrations  or  withdrawals;  to  suppress  or 
destroy  hostile  anti-ship  weapons  and  air  defense  systems;  and  to  delay  and 
disrupt  enemy  movement  and  reinforcement  of  defending  forces.  The  progr^ 
office  estimates  a  total  acquisition  cost  of  $600  million.  The  milestone  decision 
authority  for  the  program  is  the  Navy  Program  Executive  Officer  for  Surface 
Strike. 

MIDS-LVT.  The  MIDS-LVT  is  a  Navy  Acquisition  Category  ID  major 
Defense  acquisition  program  and  is  a  joint,  multi-service  cooperation 
development  program.  It  is  a  communications,  navigation,  and  identification 
system  intended  to  support  key  theater  functions  such  as  surveillance, 
identification,  air  control,  weapons  engagement  coordination,  and  direction  for 
all  Services  and  allied  forces.  The  MIDS-LVT  will  provide  jamming-resistant, 
wide-area  communications  between  the  MIDS-LVT  and  the  Joint  Tactical 
Information  Distribution  System-equipped  platforms.  It  is  intended  to  improve 
joint  interoperability,  enhance  battlefield  situation  awareness,  ensure 
information  superiority,  and  support  precision  engagement  of  threats  for  our 
forces.  Under  the  direction  of  the  Navy  Program  Executive  Officer  for  Tactical 
Air  Programs,  the  MIDS-LVT  Program  Office  estimates  a  total  program  cost  of 
$1.2  billion  in  FY  1992  base  dollars.  The  milestone  decision  authority  for  the 
program  is  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics. 
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Multiple  Launch  Rocket  System  Upgrade.  The  Multiple  Launch  Rocket 
System  Upgrade  program  is  an  Army  Acquisition  Category  IC  major  Defense 
acquisition  program.  It  satisfies  the  need  for  a  non-nuclear,  all-weather, 
indirect,  area  fire  weapon  system  to  strike  counterfire,  air  defense,  armored 
formations,  and  other  high-payoff  targets  at  all  depths  of  the  tactical  battlefield. 
The  Multiple  Launch  Rocket  System  Upgrade  consists  of  two  coniponents,  the 
guided  rocket  and  the  multiple  launch  rocket  system  launcher.  Primary  missions 
of  the  Multiple  Launch  Rocket  System  Upgrade  include  suppression, 
neutralization,  and  destruction  of  threat  fire  support  and  forward  area  air 
defense  targets.  The  Multiple  Launch  Rocket  System  Upgrade  program  office 
estimates  a  total  program  cost  of  $4.9  billion.  The  milestone  decision  authority 
for  the  program  is  the  Army  Program  Executive  Officer  for  Tactical  Missiles. 

National  Airspace  System.  The  National  Airspace  System  is  an  Air  Force 
Acquisition  Category  IC  major  Defense  acquisition  program.  It  will  modernize 
the  DoD  radar  approach  control  facilities  in  parallel  widi  the  Federal  Aviation 
Administration.  The  program  provides  systems  and  facilities  that  are 
compatible  and  interoperable  with  the  Federal  Aviation  Administration 
modernization  efforts,  prevents  DoD  flight  delays  and  cancellations,  continues 
DoD  access  into  special  use  airspace,  provides  transparent  services  to  military 
and  civil  aircraft,  replaces  aging  DoD  air  traffic  control  systems,  md  increases 
flight  safety.  The  National  Airspace  System  also  includes  the  Military  Airspace 
Management  System  that  will  schedule  and  manage  special  use  airspace.  The 
program  office  estimates  a  total  program  cost  of  $1  billion.  The  milestone 
decision  authority  for  the  program  is  the  Principal  Deputy  Assistant  Secretary  of 
the  Air  Force  for  Acquisition  and  Management. 

VIRGINIA  Class  Submarine.  The  VIRGINIA  Class  Submarine  is  a  Navy 
Acquisition  Category  ID  major  Defense  acquisition  program.  The  VIRGINIA 
Class  submarine  is  designed  to:  replace  the  LOS  ANGELES  Class  submarines; 
rapidly  deploy  to  militarily  important  hostile  ocean  areas  and  deny  their  use  to 
the  enemy;  clear  the  way  for  strikes  by  other  fi'iendly  forces;  and  engage  and 
destroy  enemy  submarines,  surface  forces  and  land  targets,  as  well  as  provide 
full-dimensional  protection  for  afloat  forces.  The  VIRGINIA  class  submarine 
program  office  estimates  a  total  program  cost  of  $65  billion.  The  milestone 
decision  authority  for  the  program  is  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics. 

Wide  Area  Munition.  The  Wide  Area  Munition  is  an  Army  Acquisition 
Category  II  major  system.  It  is  a  smart,  autonomous,  top-attack  anti-tank 
munition  designed  to  defeat  armored  combat  vehicles  from  a  standoff  distance. 

It  uses  acoustic  and  seismic  sensors  in  its  ground  platform  to  detect,  track,  and 
classify  potential  targets,  and  then  launch  an  infrared  deterring  submunition  over 
the  top  of  the  selected  target.  Once  the  submunition  detects  the  target,  it  fires 
an  explosively  formed  penetrator  to  defeat  the  target.  The  Wide  Area  Munition 
program  office  estimates  a  total  program  cost  of  $583  million.  The  milestone 
decision  authority  for  the  program  is  the  Deputy  for  Systems  Acquisition, 

U.S.  Army  Tank-automotive  and  Armaments  Command. 
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There  are  three  distinct  Wide  Area  Munition  versions.  The  Wide  Area 
Munition-basic,  the  Wide  Area  Munition  product  improvement  program-1,  and 
the  Wide  Area  Munition  product  improvement  program-2. 

Wide  Area  Munition-Basic.  The  Wide  Area  Munition-basic  is  the 
Army’s  first  generation  of  a  smart  autonomous,  top  attack  munition.  It 
includes  seismic  and  acoustic  sensors.  It  can  be  hand  emplaced  and 
either  manually  or  remotely  set. 

Wide  Area  Munition  Product  Improvement  Program-1.  This  version 
improves  upon  the  Wide  Area  Munition-basic,  increasing  radio  and  two- 
way  communication  with  improved  range. 

Wide  Area  Munition  Product  Improvement  Program-2.  This  version 
further  improves  the  weapon  system  with  active  and  passive  sublet 
sensors,  a  combined  effects  warhead,  and  ground  data  sharing. 

Wind  Corrected  Munitions  Dispenser.  The  Wind  Corrected  Munitions 
Dispenser  is  an  Air  Force  Acquisition  Category  II  major  system.  The  Air  Force 
initiated  the  program  in  response  to  operational  limitations  identified  during 
Operation  Desert  Storm.  Adverse  weather,  smoke,  dust,  and  enemy  defenses 
affected  delivery  of  direct  attack  munitions.  Cluster  munition  accuracy  was 
significantly  reduced  when  released  at  mid-to-high  altimde.  The  Wind 
Corrected  Munitions  Dispenser  consists  of  a  smart  guidance  tail  kit,  which 
replaces  the  current  tail  section  of  the  Combined  Efects  Munition,  Gator,  and 
Sensor  Fused  Weapon  weapon  systems.  The  Wind  Corrected  Munitions 
Dispenser  program  office  estimates  a  total  program  cost  of  $674  million.  The 
milestone  decision  authority  for  the  program  is  the  Principal  Deputy  Assistant 
Secretary  of  the  Air  Force  for  Acquisition  and  Management. 
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Appendix  D.  Exit  Criteria  Established  For  Nine 

Programs  Reviewed 


Listed  are  the  exit  criteria  that  milestone  decision  authorities  established  in  one 
or  more  acquisition  decision  memorandums  for  the  nine  programs  reviewed. 

For  the  nine  programs,  we  identify  the  exit  criteria  that  Ae  program  manager 
was  required  to  satisfy  before  exiting  each  applicable  acquisition  phase. 

Advanced  Extremely  High  Frequency  Program.  On  May  4,  1999,  the 
milestone  decision  authority  established  these  exit  criteria  for  exiting  the 
program  definition  and  risk  reduction  phase  of  the  acquisition  process: 

•  Provide  an  updated,  quantitative  risk  assessment  demonstrating 
program  mamrity. 

•  Successfully  complete  a  system  design  review  with  the  offeror 
demonstrating  the  potential  to  meet  performance,  schedule,  and  cost 
requirements. 

•  Conclude  a  successful  engineering  model  process  risk-reduction 
program. 

All  Source  Analysis  System.  On  July  28,  1993,  the  milestone  decision 
authority  established  these  exit  criteria  for  exiting  the  engineering  and 
manufacrnring  development  phase: 

•  During  an  initial  operational  test,  the  program  must  demonstrate  the 
capability  to  auto  generate  12,  and  auto  sanitize  2,  message  types. 

•  During  an  initial  operational  test,  the  program  must  demonstrate  the 
ability  to  provide  continuous  intelligence  support  by  resynchronizing 
databases  90  percent  of  the  time  after  tactical  displacement. 

•  The  program  must  have  three  certifications: 

-  Joint  Interoperability* 

-  Automatic  Digital  Network  Category  III 

-  Sensitive  Compartmented  Information 

•  During  an  initial  operational  test,  the  program  must  demonstrate  that 
workstation,  on-equipment  diagnostic  capabilities  permit  an  operator  to 
correctly  fault  detect  and  isolate  replaceable  units  to  a  single  line 

90  percent  of  the  time. 


*  Eidt  criteria  are  not  program  specific  as  stated  in  DoD  Regulation  5000. 2-R. 
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Extended  Range  Guided  Munitions.  On  August  5,  1996,  the  milestone 
decision  authority  established  exit  criteria  for  die  low-rate  initial  production 
decision  and  exiting  the  engineering  and  manufacturing  development  phase  of 
the  program. 

Low-rate  initial  production  decision.  A  successful  target  engagement  of 
a  land-based,  gun-fired,  extended  range  guided  munition,  in  an  operational 
requirements  document  threshold-jamming  environment. 

Engineering  and  manufacturing  development  phase.  The  successful 
completion  of  operational  evaluation  and  a  reconunendation  for  fleet 
introduction.* 

Multifunctional  Information  Distribution  System-Low  Volume  Terminal. 
On  October  5,  1994,  the  milestone  decision  authority  established  these  exit 
criteria  for  the  low-rate  initial  production  decision: 

•  Close  out  all  critical  design  review  issues.* 

•  Complete  qualification  testing.* 

•  Update  independent  cost  estimates. 

•  Revise  the  acquisition  strategy  report. 

•  Conduct  the  production  readiness  review. 

•  Verify  the  Multifunctional  Information  Distribution  System  terminal. 

•  Conduct  a  physical  configuration  audit.* 

•  Demonstrate  compatibility  and  interoperability  with  the  Joint  Tactical 
Information  Distribution  System  class-2  terminals. 

•  Include  an  early  operational  assessment  of  platform  interface  and 
operator  workload. 

Multiple  Launch  Rocket  System  Upgrade.  The  Multiple  Launch  Rocket  System 
Upgrade  consists  of  two  components,  the  guided  rocket  and  the  multiple  launch  rocket 
system  launcher  (launcher).  On  June  11,  1998,  the  milestone  decision  authority 
established  exit  criteria  for  the  guided  rocket  low-rate  initial  production  decision  and 
the  engineering  and  manufacturing  development  phase.  On  May  28,  1998,  the 
milestone  decision  authority  established  exit  criteria  for  the  launcher  engineering  and 
manufacturing  development  phase. 


*  Exit  criteria  are  not  program  specific  as  stated  in  DoD  Regulation  5000. 2-R. 
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For  the  guided  rocket  low-rate  initial  production  decision: 

•  Obtain  type  classification  with  a  successful  functional  configuration  audit. 

•  Obtain  an  88. 1 1  percent  reliability  rate. 

•  Complete  a  successful  production  readiness  review. 

For  the  guided  rocket  engineering  and  manufacturing  development  phase: 

•  The  guided  rocket  must  have  a  maximum  range  of  60  kilometers  and  a 
minimum  range  of  15  kilometers. 

•  The  guided  rocket  must  have  an  effectiveness  rate  of  30  percent  expected 
fractional  damage. 

•  Guided  rocket  reliability  must  be  86.7  percent. 

•  There  must  be  a  less  than  1  percent  hazardous  dud  rate. 

For  the  launcher  engineering  and  manufacturing  development  phase: 

•  To  fire  each  t)^e  of  the  Multiple  Launch  Rocket  System  family  of  missiles 
that  are  in  production. 

National  Airspace  System.  The  milestone  decision  authority  established  exit  criteria 
for  exiting  the  program  definition  and  risk  reduction  phase  and  the  engineering  and 
manufacturing  development  phase  on  November  5,  1992,  and  July  24,  1995, 
respectively. 

For  the  program  definition  and  risk  reduction  phase: 

•  Demonstrate  repackaging  the  Federal  Aviation  Administration  Common 
Console  into  the  DoD  configuration. 

•  Demonstrate  Military  Airspace  Management  System  prototype. 

•  Sign  executive  interagency  agreements  for  test,  procurement,  and  support  of 
Federal  Aviation  Administration  Advanced  Automation  System.* 

For  the  engineering  and  manufacturing  development  phase: 

•  Award  the  nondevelopmental  items  contracts  for  the  Voice  Communications 
Switching  System,  Digital  Airj^ort  Surveillance  Radar,  and  Digital  Advanced 
Automation  System  programs. 

•  Successfully  complete  the  DoD  National  Airspace  System  Initial  Operational 
Test  and  Evaluation.  * 


’  Exit  criteria  are  not  program  specific  as  stated  in  DoD  Regulation  5000. 2-R. 
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•  Incorporate  the  Major  Range  and  Test  Facility  Base  and  Mainteiiance 
Replacement  Unit  quantities  into  the  National  Airspace  System,  if 
quantities  and  funding  are  provided  in  time  for  the  option  exercise. 

VIRGINIA  Class  Submarine.  The  milestone  decision  authority  established  exit 
criteria  for  exiting  the  program  definition  and  risk  reduction  phase  and  the  low- 
rate  initial  production  decision  on  August  18,  1994,  and  June  30,  1995, 
respectively. 

For  the  program  definition  and  risk  reduction  phase: 

•  Ensure  that  the  Navy  operational  test  agency  completes  an  early 
operational  assessment  on  the  potential  of  the  preliminary  design  to 
satisfy  the  requirements  of  the  operational  requirements  document.* 

•  Document  the  preliminary  design  in  an  approved  ship  specification.* 

•  Weight  estimate  the  ship  design. 

•  Estimate  the  inboard  electronics  configiuation. 

•  Ensure  that  the  OSD  staff  review  the  lead^ship  design  and  construction 
“Request  for  Proposal”  prior  to  issuance.* 

•  Submit  a  waiver  from  full-up  system  level  live-fire  test  and  evaluation 
for  approval.* 

For  the  low-rate  initial  production  decision: 

•  The  independent  test  organization  for  the  Navy  must  update  the 
operational  assessment. 

•  The  Navy  must  provide  a  report  that  assesses  the  vulnerability  of  the 
ship  in  accordance  with  the  live-fire  test  and  evaluation  plan. 

•  The  Director,  Strategic  and  Tactical  Systems,  must  estoblish  an 
integrated  product  team  to  review  the  ship  and  the  design  maturity  of 
the  command,  control,  communication,  and  intelligence  system  to 
determine  their  adequacy  to  proceed  with  the  lead  ship  construction, 
and  report  the  results  of  the  review. 

Wide  Area  Munition.  The  Wide  Area  Munition  consists  of  a  basic  version  and 
two  product  improvement  program  versions.  On  June  18,  1996,  the  milestone 
decision  authority  established  exit  criteria  for  the  low-rate  initial  production 
decision  for  the  Wide  Area  Munition-basic,  and  exit  criteria  for  product 
improvement  program- 1  that  the  milestone  decision  authority  did  not  stipulate  to 
a  specific  decision  point  or  acquisition  phase. 


*  Exit  criteria  are  not  program  specific  as  stated  in  DoD  Regulation  5000. 2-R. 
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For  the  basic  version  low-rate  initial  production  decision: 

•  Unit  cost  shall  not  exceed  contract  target  cost  by  more  than  5  percent. 

•  Technical  test  hardware  built  on  production  tooling  will  demonstrate 
lead  times,  production  processes,  production  quality,  and  reliability 
within  5  percent  of  predicted  learning  curves. 

•  The  system  performance  evaluation  will  demonstrate  that,  using 
mission  employment  practices,  performance  specified  in  the 
operational  requirement  document  can  be  achieved.* 

For  the  product  improvement  program- 1: 

•  The  test  program  shall  demonstrate  that  the  communication,  control, 
and  redeployability  functions  shall  be  achieved  as  specified  when  used 
in  accordance  with  mission  employment  practices. 

•  The  production  readiness  review  supports  transition  to  fiill-rate 
production  and  validates  production  quality  and  processes.* 

Wind  Corrected  Munition  Dispenser.  On  September  29,  1994,  the  milestone 
decision  authority  established  exit  criteria  for  the  first  low-rate  initial  production 
decisions.  On  August  3,  1998,  established  exit  criteria  for  the  second  low-rate 
initial  production  decision  and  for  exiting  the  engineering  and  manufacturing 
development  phase.  On  July  20,  1999,  the  milestone  decision  authority 
established  exit  criteria  for  Ae  thdrd  low-rate  initial  production  decision. 

For  the  first  low-rate  initial  production  decision: 

•  Demonstrate  operation  from  medium-to-high  altitude. 

•  Average  unit  cost  procurement  price  must  be  within  objectives. 

For  the  second  low-rate  initial  production  decision: 

•  Complete  margin  and  qualification  testing. 

•  Complete  end-to-end  testing. 

•  Complete  at  least  one  successful  flight  with  pin-puller. 

For  the  third  low-rate  initial  production  decision: 

•  Complete  combined  developmental  and  operational  tests. 

•  Complete  Block  3A  qualification  testing. 

•  Average  unit  procurement  price  must  be  within  objectives. 


‘  Exit  criteria  are  not  program  specific  as  illustrated  in  DoD  Regulation  5000.2-R. 
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For  the  engineering  and  manufacturing  development  phase: 

•  Complete  dedicated  operational  tests  on  B-52  and  F-16  aircraft. 

•  Meet  the  acquisition  program  baseline  thresholds.* 

•  The  contractor  must  meet  full-rate  production  ramp  requirements. 

•  Achieve  average  unit  procurement  price  within  objectives. 

•  Submit  required  documents.* 


’  Exit  criteria  are  not  program  specific  as  illustrated  in  DoD  Regulation  50(X).2-R. 
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Deputy  For  Systems  Acquisition,  U.S.  Army 
Tank-automotive  and  Armaments  Command 
Comments 


DEPARTMENT  OF  THE  ARMY 

UNrTH3  STATES  ARMY  TANK-AUTOMOTIVE  AND  ARMAMENTS  COMMAND 
WARREN.  MICHIGAN  48397-5000 


RSPtrTO 

ATTENTtONW 

AMSTA-CM-PA  (36-2b) 


.  ^  DEC  20^ 


MEMORANDUM  FOR  Cotnoiander#  U.S.  Ariny  Materiel  Command,  ATTN:  AMCIR-A, 
£001  Eisenhower  Avenue,  Alexandria,  VA  22333-0001 

SUBiJSCT;  1ACX5M  Reply  to  DoDIG  Draft  Report:,  Use  of  Exit  Criteria  for 
Major  Defense  Systems  {AMC  No.  DOOlO) 


X.  Reference  memorandum,  HQ  AMC,  AMCIR-A,  17  October  2000,  SAB. 

2.  In  accordance  with  AR  36-2,  we  are  attaching  the  TACOM  position  to 
the  subject  draft  report-  The  reply  contains  additicaial  facts  for  the 
auditors'  consideration.  Also,  due  to  the  recent  restructuring  of  the 
program,  we  agree  that  revised  exit  criteria  is  needed.  Our  target 
date  for  this  revised  exit  criteria  is  contained  in  the  reply . 

3.  The  TACOM  POC  for  this  audit  is  Chuck  Krulic,  DSN  786-6158,  or  by 
e-mail  at  krulic c4&tacoiti.  army  .mil, . 


Atch 

Chief  of  Staff 


! 

i 


31 


FINDING  A:  Establishing  Program-Specific  Exit  Criteria 


For  eight  of  the  nine  programs  reviewed,  milestone  decision  authorities  did  not  ensure 
that  the  program  managers  proposed  program-specific  exit  criteria  for  use  at  future 
milestone  decision  point(s).  TMs  condition  occurred  because: 

•  The  milestone  decision  authorities  for  four  progiams  did  not  enforce  the 
requirement  that  program  managers  propose  exit  criteria  tor  use  at  future 
milestone  decision  pomt(s),  and 

•  Program  managers  for  eigjit  of  Ihe  nine  programs  proposed  one  or  more 
exit  criteria  that  repeated  the  mmlTniim  requiicmciits  for  acquisition 
nrilestone  phases  contained  in  the  approved  program  baseline  and  DoD 
Regularion  5000.2-R. 

As  a  result,  the  milestone  decision  audiorities  were  limited  in  fiieir  ability  to  use  exit 
criteria  as  a  management  tool  to  determine  whether  programs  under  theor  review  and 
oversight  should  progress  within  an  acquisition  phase  or  continue  Into  the  next 
acqtiisition  phase  at  milestone  decision  points. 

Additional  Facts:  The  following  cominents  are  ofGared  as  clarification  to  the  report: 

(a)  The  executive  summary  states  that  the  DSA  needs  to  establish  exit  criteria. 
This  inajqjropiiatcly  implies  thai  exit  criteria  did  not  exist  Request  that  you 
modity  die  snnnmary  to  reflect  your  teams  assessment  that  die  criteria  needs  to 
be  updated  based  on  the  recently  restructured  program.  The  error  seems  to 
stem  from  a  conflict  between  the  assessments  on  pages  6  and  7  of  Ihe  report 
stating  that  criteria  was  not  established,  which  is  in  error,  and  in  conflict  widi 
the  criteria  stated  in  the  MDA  and  refcrftttced  on  page  25  of  the  report.  This 
will  also  require  a  modificadon  to  recomm^idation  A-2  for  the  same  reason. 

(b)  The  basic  WAM  is  currently  in  LRIP  and  there  is  no  intent  to  go  to  a  full  rate 
produedon  decision.  As  mentioned  in  the  report,  the  WAM  PEP  program  is 
currently  being  restructured  into  a  single  program  called  the  Advanced  WAM. 
Revised  exit  criteria  for  entering  LRIP  and  Full  Rate  Production  for  the 
Advanced  WAM  is  currently  being  developed. 


ytcKommendlation  A-2;  We  recommend  that  the  Deputy  for  Systems  Acquisition,  U.S. 
Army  Tank-automotive  and  Armaments  Command  establish  todt  criteria  for  the  low-rate 
initii  production  decision  and  the  engineeting  and  manufocturiug  development  phase  for 
Ihe  product  improvement  programs-l  and  -2  of  the  Wide  Area  Munition. 
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Space  And  Naval  Warfare  Systems  Command 
Comments 


DePARTMENT  OF  THE  NAVY 

SIWXAW 

«M1MCVKMQK(nir 
SANMdO,CAB110-»19 

7500 

Ser00<3/0$5 

DEC  8  2000 

From;  Commander,  Space  and  Naval  Wai&re  Systems  Command 
To:  Department  of  Defense  Office  of  the  Inspector  General 

Subj:  USEOFEXITCRITERIAFORMAJORDEFENSESYSTEMS(OAE-0121) 

Ref  (a)  DoDIG  Memo  of  1 1  Oct  00 

(b)  DoDIG  Draft  of  Proposed  Audit  Report  Dated  1 1  Oct  00 

(1)  Space  and  Naval  Warfare  Systems  Command  (SPAWAR)  Comments  to  Department 
of  Defense  Office  of  the  Inspector  G^wral  Draft  Audit  Report  #0AE-0121  “Use  of 
Exit  Criteria  for  M^or  De^e  Systems" 

1 .  Per  reference  (a),  comments  to  the  draft  report  contained  in  reference  (b)  are  provided  in 
endosuTB  (1).  Wc  do  not  concur  with  the  audit  findings  applicable  to  SPAWAR  and  our  written 
conmients  address  riiese  findings. 

2.  Questions  reading  this  response  may  be  directed  to  John  Gampel,  Deputy  Inspector 
General,  at  (519)  524-7065  or  DSN  524-7065,  or  Margaret  Gordon,  Audit  Liaison  point  of 
contact,  at  (619)  524-7075  or  DSN  524-7075. 


K.  D.  Slaght 

Rear  Admiral,  D.S.  Navy 
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Space  and  Naval  Warfare  ^$tems  Conunasid  (SPAWAB) 
Comments  to  ]>epartinent  of  Defense 
Office  of  the  Inspector  General  Draft  Audit  Deport  #OaS-0121 
Use  of  Esdt  Criteria  for  Major  Defense  Systems 


Establishing  Program-SpeciJic  Exit  Criteria 

"For  right  of  the  nin©  programs  reviewed,  milestone  derision  authorities  did  not  ensure  dial  the 
program  managers  proposed  progam-spedfic  exit  criteria  for  use  at  future  milestone  deasion 
pOfint(s)” 

i .  Appendix  D.  Exit  Criteria  Established  for  Nine  Programs  Reviewed 

^^MnltifaiLctional  Information  Distribntioii  SystentLow  Vohune  TenninaL 
On  October  S,  1994,  the  milestone  deririon  authority  established  these  exit 
critoia  for  die  low*  rate  initial  prodoctiem  derision; 

•  Close  out  all  critical  design  review  issues.* 

••  Complete  qualification  testing.* 

•  Update  indqiendeixt  cost  estimates. 

•  Revise  the  acquisition  strategy  rq»rt 

•  Conduct  the  production  readiness  review.* 

•  Verify  the  MultifimctioDal  hiformatimi  Distribution  System  terminal. 

•  Conduct  a  physical  coufiguration  audit.* 

•  Demonstrate  compatibility  and  Inteiopersibilify  witii  the  Joint  Tactical  hifannalion 
Distribution  Syst^  class-2  terminals. 

•  Inriude  an  early  operational  assessmttt  of  platform  inter&ce  and  operator 
woridoad. 

*  Exit  criteria  are  not  program-specific  as  stated  in  DoD  Regulation  5000.2-R-” 


Eiiddsored} 
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The  nme  Mdtifunctional  Infwmalion  Distribation  System  (MIDS)  eodt  ciitcda  for 
entering  the  low-rate  initial  pcoduction  (LRIP),  as  stated  in  Appendix  D  of  the  audit 
rqxnl;  are  program,  spedficx  Those  idortified  in  the  audit  report  as  not  program-specific 
are  diiecdy  related  to  die  program  execution.  Critical  design  review  issaes.  convicted 
quaHfication  testing^  the  production  readiness  review,  and  a  physical  configuration  audit 
are  all  directiy  program-^oecific  and  provided  tiie  milestone  decision  authority  with 
in&imation  to  make  the  iq^propriate  programmatic  decision. 


2.  Establishing  Exit  Criteria  on  die  Nine  Programs  Reviewed 
The  table  on  Page  6  of  the  report  states  that  tiie  Multifiiiictioiial  Information  Distribution 
Sh^stem-Low  Volume  Terminal  (MIDS-LVT)  program  milestone  decision  authority  did 
not  est^Ush  exit  criteria  for  the  engmeering  and  manu&cturing  devdopment  phase. 

ThehdlDS  program  had  nine  ^ecific  exit  crit^  for  LRIP  decision  point  (April  00)  as 
stated  in  Appendix  D  of  the  audit  r^Kut  These  nine  exit  criteria  were  predicated  on  the 
successfid  accomplishment  of  efforts  completed  during  the  Bngineering  and 
Manufoctoting  Development  (EMD)  phase  of  tiie  program.  The  EMD  MIDS  terminal 
contractor  completed  its  contractual  effort  in  Jane  00,  A  MIDS  systems  engmeering  and 
int^iration  ocmtiact  is  in  place  to  support  the  HMD  platform,  integration  phase  of  the 
IKOgram.  The  MIDS  pocogram  has  specific  exit  criteria  tied  directly  to  platform 
integration  testing  for  LRIP  2  decision  point  (April  OIX  as  established  by  the  milestone 
deosion  authority  at  the  last  derision  point  (LRJP  April  00).  The  program  is 

cunentiy  coordinating  tije  exit  criteria  fwn^estone  IH  (entry  into  full  rate  prodoriion) 
with  the  MIDS  Overarrinng  Integrated  Product  Team  (OIPI).  The  exit  criteria  resulting 
firam  tins  effort  will  be  submitted  to  tile  milestone  derision  authority  for  i^proval. 
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